Drilling fluids are used in oil and gas drilling to assist with lubricating the drill bit, ensuring well safety, forming filter cakes to minimize fluid loss into drilling formations, and transporting rock debris to the surface of the well. Invert emulsion drilling fluids, which are emulsions of water in an oil or synthetic base phase, are used in some drilling environments, such as in wells having features that may react to water-based drilling fluids or in high temperature or high pressure environments. Clays are sometimes added to invert emulsion fluids to act as viscosifiers and to stabilize the water-in-oil emulsion of the drilling fluid. In some instances, low gravity solids, such as small calcium carbonate particles or clay-type materials, are added to invert emulsion drilling fluids instead of clay to provide improved thermal stability, resistance to high pressure, and tolerance to contamination.